Repository navigation
feat(data): point-in-time fundamental fields (growth / valuation / quality) - #27
Open
agenthaulk wants to merge 4 commits into
Open
agenthaulk wants to merge 4 commits into
agenthaulk wants to merge 4 commits into
Conversation
Wire the mounted lakehouse's fundamental source layer through to the workbench so an idea like 年报季报利润同比 parses to a real rank(netprofit_yoy) factor instead of a nearest price-volume approximation. - data/fundamentals.py (public, path-free): reads Tushare-Pro-schema financial-statement parquet and expands each report to a daily point-in-time value via an ann_date as-of join -- a figure is visible only from its announcement date, never the period end, and never a report announced in the future (no look-ahead). Daily valuation ratios (daily_basic) merge on trade_date. 24 exposed fields across growth / profitability / valuation / leverage / cashflow. - data/local.py: the field catalog gains the fundamental fields (single spec table -> catalog -> validation -> MCP/LLM surface); the panel loader left-merges a PIT overlay (fundamentals.parquet sibling or configured root); validate() is overlay-aware so availability is honest (FP-4). - config: optional fundamentals_source_root / fundamentals_overlay_root (both None -> price-volume-only, unchanged). Real paths live only in an ignored local config; no private path or provider name enters tracked code. - CLI qf data build-fundamentals materializes the overlay from the source layer. LLM prompt gains fundamental formula hints; the out-of-scope honesty warning narrows to genuinely-unavailable alt-data (sentiment / analyst / northbound), since fundamentals are now expressible. Demo workspace ships a synthetic overlay + a rank(netprofit_yoy) demo factor. - tests: PIT look-ahead correctness (ann_date semantics, no future report, restatement supersedes only from its announcement), overlay merge, catalog expansion; existing surface tests updated for the added fields/factor. release_safety_scan passes; full suite 2001 passed.
_runtime_paths_from_config rebuilt PathSettings but dropped fundamentals_source_root/fundamentals_overlay_root, so 'qf data build-fundamentals' reported the source root unconfigured even when the config set it. Carry both fields through; add an end-to-end CLI regression that materializes an overlay from a synthetic source via --config. Found by real-mounted-drive acceptance (materialized the live A-share fundamentals overlay in 7.9s over 3.19M rows; PIT spot-check confirmed value_at(D) = latest report announced on-or-before D, future announcements excluded).
Opus review confirmed the PIT look-ahead property holds; fixes for the rest: - portability: force all merge_asof join keys to datetime64[ns] (pandas>=2.2 rejects a us/ns mismatch that pandas 3.x tolerated) -- caught by the container build. - robustness: _parse_ymd coerces through nullable Int64 so a NaN-bearing int date column (which pandas upcasts to float -> '20250420.0') still parses instead of silently NaT-ing every row; as_of_expand guards the empty-reports case instead of crashing merge_asof on a dtype mismatch. - remove the dead fundamentals_overlay_root config: it was never wired to any panel reader (only the CLI writer), so a configured overlay was invisible. Sibling fundamentals.parquet is the single discovery path; CLI --output overrides. - loader de-dups overlay (trade_date,instrument) keys so a malformed overlay can't fan out the panel (its FP-4 contract). - nl_flow keeps out-of-scope terms for balance-sheet line items with no exposed catalog field (goodwill/inventory/receivable/商誉/存货/应收). - vendor-neutral wording (documented open schema, not a vendor name). - document the restatement modeling choice (latest announcement, no look-ahead). Tests added for the float/NaN date parse, empty-reports guard, and the overlay duplicate-key fan-out.
Separate canonical field support from fields actually backed by the configured data root. Validate optional fundamental overlays before advertising or merging them, and reject dependent formulas before execution when data is absent, malformed, duplicated, unmatched, or entirely null. Project source parquet reads from schema metadata so files are not loaded fully twice. Constraint: Price-volume research must remain usable when optional fundamentals are absent or invalid Rejected: Silently deduplicate or ignore malformed fundamental data | hides data-quality failures from dependent formulas Confidence: high Scope-risk: moderate Directive: Keep capability catalogs distinct from runtime field availability Tested: 2017 pytest tests; release safety scan; compileall; diff check; independent code review Not-tested: Real mounted-drive fundamental overlay rebuild
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wires the mounted lakehouse's fundamental source layer through to the workbench so a fundamentals idea — e.g. "年报、季报利润同比增加高的公司，未来21天更好" — parses to a real
rank(netprofit_yoy)factor instead of the nearest price-volume approximation it produced before.The raw data was already on the drive (
source_snapshot/.../{income,financial,daily_basic,...}withann_dateannouncement dates); only the pipeline was missing. This adds it as a layered, point-in-time-correct data plane.What's added
data/fundamentals.py(public, path-free) — reads Tushare-Pro-schema statement parquet and expands each report to a daily point-in-time value via anann_dateas-of join: a figure is visible only from its announcement date, never the report period end, and never a report announced in the future (no look-ahead). Daily valuation ratios (daily_basic) merge ontrade_date. 24 exposed fields across growth / profitability / valuation / leverage / cashflow (netprofit_yoy,or_yoy,roe,netprofit_margin,pe_ttm,pb,debt_to_assets, …).data/local.py— the field catalog gains the fundamental fields from a single spec table (→ validation → MCP/LLM advertised surface, one edit propagates everywhere); the panel loader left-merges a PIT overlay (fundamentals.parquetsibling or a configured root);validate()is overlay-aware so the data-catalog availability is honest (FP-4, never a silent 0).fundamentals_source_root/fundamentals_overlay_root(bothNone→ price-volume-only, unchanged). Real paths live only in an ignored local config; no private path or provider name enters tracked code.qf data build-fundamentals— materializes the overlay from the source layer.rank(netprofit_yoy)demo factor, so the field route works without the mounted drive.OSS / policy
The processing functions are public; the source layout (a Tushare-Pro open schema) is a documented default; the mounted path lives only in a git-ignored local config. No network provider or ingestion client is bundled (consistent with the existing local source-snapshot loader precedent).
release_safety_scan.pypasses on 280 public files with zero private paths.Tests / status
tests/test_fundamentals_pit.pypins the look-ahead correctness: value usesann_datenot period end; never pulls a future report; a restatement supersedes only from its own announcement; same-day dedup keeps the latest period.rank(netprofit_yoy)factor evaluates end-to-end.Relationship to the open stack
Stacked on #26 (honesty-warning) — this PR's §nl_flow narrows #26's out-of-scope terms now that fundamentals are supported. #26 in turn sits on #25 (integration docs). Recommended merge order: #25 → #26 → this. Real-mounted-drive acceptance (
qf data build-fundamentalsover the live source layer + browser reproduction) and an adversarial PIT review are being run as the final acceptance step.